postgresql


Possible to use pandas/sqlalchemy to insert arrays into sql database? (postgres)


With the following:
engine = sqlalchemy.create_engine(url)
df = pd.DataFrame({
"eid": [1,2],
"f_i": [123, 1231],
"f_i_arr": [[123], [0]],
"f_53": ["2013/12/1","2013/12/1",],
"f_53a": [["2013/12/1"], ["2013/12/1"],],
})
with engine.connect() as con:
con.execute("""
DROP TABLE IF EXISTS public.test;
CREATE TABLE public.test
(
eid integer NOT NULL,
f_i INTEGER NULL,
f_i_arr INTEGER NULL,
f_53 DATE NULL,
f_53a DATE[] NULL,
PRIMARY KEY(eid)
);;
""")
df.to_sql("test", con, if_exists='append')
If I try to insert only column "f_53" (an date) it succeeds.
If I try to add column "f_53a" (a date[]) it fails with:
^
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "f_53a" is of type date[] but expression is of type text[]
LINE 1: ..._53, f_53a, f_i, f_i_arr) VALUES (1, '2013/12/1', ARRAY['201...
^
HINT: You will need to rewrite or cast the expression.
[SQL: 'INSERT INTO test (eid, f_53, f_53a, f_i, f_i_arr) VALUES (%(eid)s, %(f_53)s, %(f_53a)s, %(f_i)s, %(f_i_arr)s)'] [parameters: ({'f_53': '2013/12/1', 'f_53a': ['2013/12/1', '2013/12/1'], 'f_i_arr': [123], 'eid': 1, 'f_i': 123}, {'f_53': '2013/12/1', 'f_53a': ['2013/12/1', '2013/12/1'], 'f_i_arr': [0], 'eid': 2, 'f_i': 1231})]
Yes -- is possible to insert [] and [][] types from a dataframe into postgres form a dataframe.
Unlike flat DATE types, which are may be correctly parsed by sql, DATE[] and DATE[][] need to be converted to datetime objects first. Like so.
with engine.connect() as con:
con.execute("""
DROP TABLE IF EXISTS public.test;
CREATE TABLE public.test
(
eid integer NOT NULL,
f_i INTEGER NULL,
f_ia INTEGER[] NULL,
f_iaa INTEGER[][] NULL,
f_d DATE NULL,
f_da DATE[] NULL,
f_daa DATE[][] NULL,
PRIMARY KEY(eid)
);
""")
d = pd.to_datetime("2013/12/1")
i = 99
df = pd.DataFrame({
"eid": [1,2],
"f_i": [i,i],
"f_ia": [None, [i,i]],
"f_iaa": [[[i,i],[i,i]], None],
"f_d": [d,d],
"f_da": [[d,d],None],
"f_daa": [[[d,d],[d,d]],None],
})
df.to_sql("test", con, if_exists='append', index=None)

Related Links

Connection lost after query runs for few minutes in PostgreSQL
South database error: relation already exists
How to get the Index column order(ASC, DESC, NULLS FIRST…) from Postgresql?
CakePHP get full timestamp from PostgreSQL
Postgresql server instances
delete duplicate rows in large postgresql database table
Postgresql extension: edit the source code (pgRouting, for example)
how to use composite data types (e.g. geomval) in SQLAlchemy?
No postgresql user created after new Windows install
Is there a logically equivalent and efficient version of this query without using a CTE?
Why are values copied from returned grid marked with quotes?
PL/pgSQL - insert multiple rows, returning the ids in a resultset
Postgres autovacuum fails to start due to misconfiguration
Error creating PLTCL language on binary windows (7) 64 bit Enterprise DB install
PostgreSQL how to pass more than 100 arguments to a function
Prediction avoiding landmass

Categories

HOME
drupal
kendo-ui
fortran
ldap
wifi
popover
readdir
cryengine
nltk
spss
x509certificate
azure-resource-manager
pandoc
ms-application-insights
download
css-selectors
browser-cache
autocad
kendo-asp.net-mvc
country-codes
pyodbc
angularjs-ng-repeat
ios-universal-links
rename
minimax
activejdbc
distinct
decoding
windows-phone
openam
virtuemart
orange
autologin
bootstrap-datetimepicker
serenity-js
dev-c++
beaker-notebook
infinite-loop
oracle-golden-gate
git-submodules
pushpad
node-soap
jsonapi-resources
javax
hippocms
scip
lfe
server-side-swift
autodesk-data-management
uitextview
datagridviewcombobox
binary-decision-diagram
inner-join
nao-robot
history
lucee
runtime-permissions
richtext
webmatrix
eclipse-mars
dbi
x12
falcon
airwatch
django-taggit
database-administration
mod
trendline
music-player
apache2-module
ewam
web-performance
transport
bootstrap-tabs
textblob
maven-archetype
pysvn
aiff
kss
xmlserializer
episerver-7
iwork
internet-radio
acrofields
carrier
nomachine
mysql-error-1146
sygic
xrandr
usersettings
backbone-views
panelgrid
outlook-object-model
codeigniter-form-helper
systrace
nserror
project-files
ruby-1.9
xpolog
llblgen
datarelation
nscharacterset
iphone-maps
httpbrowsercapabilities
scalable

Resources

Encrypt Message



code
soft
python
ios
c
html
jquery
cloud
mobile