Pandas · Reading and writing files

Most data arrives as files. The read_csv parameters I’ll use most:

• sep: the delimiter (",", ";", "\t")
• usecols: load only the columns you need
• dtype: fix column types up front
• parse_dates: turn date columns into datetime64
• na_values: extra strings that mean “missing”

When writing, to_csv(path, index=False). Otherwise the index becomes an extra unnamed column the next time you read the file. to_json and read_json exist too, as does Parquet (a typed, compressed column format) if pyarrow is installed.

The snippet reads from an in-memory string so it runs anywhere.

import io
import pandas as pd

raw = """date;city;temp_c;rain_mm
2024-03-01;Gandhinagar;31.2;0
2024-03-02;Gandhinagar;32.5;-
2024-03-01;Ahmedabad;32.0;0.4
2024-03-02;Ahmedabad;33.1;0
"""
df = pd.read_csv(io.StringIO(raw), sep=";", parse_dates=["date"], na_values=["-"],
                 dtype={"city": "category"})
print(df)
print(df.dtypes)

buf = io.StringIO()
df.to_csv(buf, index=False)
print(buf.getvalue().splitlines()[:2])

back = pd.read_csv(io.StringIO(buf.getvalue()))
print(back.shape, "columns:", list(back.columns))
print(df.to_json(orient="records", date_format="iso")[:80], "...")

Output:

        date         city  temp_c  rain_mm
0 2024-03-01  Gandhinagar    31.2      0.0
1 2024-03-02  Gandhinagar    32.5      NaN
2 2024-03-01    Ahmedabad    32.0      0.4
3 2024-03-02    Ahmedabad    33.1      0.0
date       datetime64[us]
city             category
temp_c            float64
rain_mm           float64
dtype: object
['date,city,temp_c,rain_mm', '2024-03-01,Gandhinagar,31.2,0.0']
(4, 4) columns: ['date', 'city', 'temp_c', 'rain_mm']
[{"date":"2024-03-01T00:00:00.000","city":"Gandhinagar","temp_c":31.2,"rain_mm": ...