postgresql
Possible to use pandas/sqlalchemy to insert arrays into sql database? (postgres)
With the following: engine = sqlalchemy.create_engine(url) df = pd.DataFrame({ "eid": [1,2], "f_i": [123, 1231], "f_i_arr": [[123], [0]], "f_53": ["2013/12/1","2013/12/1",], "f_53a": [["2013/12/1"], ["2013/12/1"],], }) with engine.connect() as con: con.execute(""" DROP TABLE IF EXISTS public.test; CREATE TABLE public.test ( eid integer NOT NULL, f_i INTEGER NULL, f_i_arr INTEGER NULL, f_53 DATE NULL, f_53a DATE[] NULL, PRIMARY KEY(eid) );; """) df.to_sql("test", con, if_exists='append') If I try to insert only column "f_53" (an date) it succeeds. If I try to add column "f_53a" (a date[]) it fails with: ^ sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "f_53a" is of type date[] but expression is of type text[] LINE 1: ..._53, f_53a, f_i, f_i_arr) VALUES (1, '2013/12/1', ARRAY['201... ^ HINT: You will need to rewrite or cast the expression. [SQL: 'INSERT INTO test (eid, f_53, f_53a, f_i, f_i_arr) VALUES (%(eid)s, %(f_53)s, %(f_53a)s, %(f_i)s, %(f_i_arr)s)'] [parameters: ({'f_53': '2013/12/1', 'f_53a': ['2013/12/1', '2013/12/1'], 'f_i_arr': [123], 'eid': 1, 'f_i': 123}, {'f_53': '2013/12/1', 'f_53a': ['2013/12/1', '2013/12/1'], 'f_i_arr': [0], 'eid': 2, 'f_i': 1231})]
Yes -- is possible to insert [] and [][] types from a dataframe into postgres form a dataframe. Unlike flat DATE types, which are may be correctly parsed by sql, DATE[] and DATE[][] need to be converted to datetime objects first. Like so. with engine.connect() as con: con.execute(""" DROP TABLE IF EXISTS public.test; CREATE TABLE public.test ( eid integer NOT NULL, f_i INTEGER NULL, f_ia INTEGER[] NULL, f_iaa INTEGER[][] NULL, f_d DATE NULL, f_da DATE[] NULL, f_daa DATE[][] NULL, PRIMARY KEY(eid) ); """) d = pd.to_datetime("2013/12/1") i = 99 df = pd.DataFrame({ "eid": [1,2], "f_i": [i,i], "f_ia": [None, [i,i]], "f_iaa": [[[i,i],[i,i]], None], "f_d": [d,d], "f_da": [[d,d],None], "f_daa": [[[d,d],[d,d]],None], }) df.to_sql("test", con, if_exists='append', index=None)
Related Links
Connection lost after query runs for few minutes in PostgreSQL
South database error: relation already exists
How to get the Index column order(ASC, DESC, NULLS FIRST…) from Postgresql?
CakePHP get full timestamp from PostgreSQL
Postgresql server instances
delete duplicate rows in large postgresql database table
Postgresql extension: edit the source code (pgRouting, for example)
how to use composite data types (e.g. geomval) in SQLAlchemy?
No postgresql user created after new Windows install
Is there a logically equivalent and efficient version of this query without using a CTE?
Why are values copied from returned grid marked with quotes?
PL/pgSQL - insert multiple rows, returning the ids in a resultset
Postgres autovacuum fails to start due to misconfiguration
Error creating PLTCL language on binary windows (7) 64 bit Enterprise DB install
PostgreSQL how to pass more than 100 arguments to a function
Prediction avoiding landmass