I have a dataframe with unix times and prices in it. I want to convert the index column so that it shows in human readable dates.
So for instance I have date as 1349633705 in the index column but I’d want it to show as 10/07/2012 (or at least 10/07/2012 18:15).
For some context, here is the code I’m working with and what I’ve tried already:
import json
import urllib2
from datetime import datetime
response = urllib2.urlopen('http://blockchain.info/charts/market-price?&format=json')
data = json.load(response)
df = DataFrame(data['values'])
df.columns = ["date","price"]
#convert dates
df.date = df.date.apply(lambda d: datetime.strptime(d, "%Y-%m-%d"))
df.index = df.date
As you can see I’m using
df.date = df.date.apply(lambda d: datetime.strptime(d, "%Y-%m-%d")) here which doesn’t work since I’m working with integers, not strings. I think I need to use datetime.date.fromtimestamp but I’m not quite sure how to apply this to the whole of df.date.
Thanks.
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
These appear to be seconds since epoch.
In [20]: df = DataFrame(data['values'])
In [21]: df.columns = ["date","price"]
In [22]: df
Out[22]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 358 entries, 0 to 357
Data columns (total 2 columns):
date 358 non-null values
price 358 non-null values
dtypes: float64(1), int64(1)
In [23]: df.head()
Out[23]:
date price
0 1349720105 12.08
1 1349806505 12.35
2 1349892905 12.15
3 1349979305 12.19
4 1350065705 12.15
In [25]: df['date'] = pd.to_datetime(df['date'],unit='s')
In [26]: df.head()
Out[26]:
date price
0 2012-10-08 18:15:05 12.08
1 2012-10-09 18:15:05 12.35
2 2012-10-10 18:15:05 12.15
3 2012-10-11 18:15:05 12.19
4 2012-10-12 18:15:05 12.15
In [27]: df.dtypes
Out[27]:
date datetime64[ns]
price float64
dtype: object
Method 2
If you try using:
df[DATE_FIELD]=(pd.to_datetime(df[DATE_FIELD],***unit='s'***))
and receive an error :
“pandas.tslib.OutOfBoundsDatetime: cannot convert input with unit ‘s’”
This means the DATE_FIELD is not specified in seconds.
In my case, it was milli seconds – EPOCH time.
The conversion worked using below:
df[DATE_FIELD]=(pd.to_datetime(df[DATE_FIELD],unit='ms'))
Method 3
Assuming we imported pandas as pd and df is our dataframe
pd.to_datetime(df['date'], unit='s')
works for me.
Method 4
The Pandas Documentation gives this and other format examples and wasn’t included in any of the above previous answers. Link:
https://pandas.pydata.org/docs/reference/api/pandas.to_datetime.html
Code
pd.to_datetime(1490195805, unit='s')
Timestamp(‘2017-03-22 15:16:45’)
pd.to_datetime(1490195805433502912, unit='ns')
Timestamp(‘2017-03-22 15:16:45.433502912’)
Method 5
Alternatively, by changing a line of the above code:
# df.date = df.date.apply(lambda d: datetime.strptime(d, "%Y-%m-%d"))
df.date = df.date.apply(lambda d: datetime.datetime.fromtimestamp(int(d)).strftime('%Y-%m-%d'))
It should also work.
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0