cca*_*123 5 python permissions connector database-schema snowflake-cloud-data-platform
我正在编写一个 python 脚本，该脚本旨在处理一些数据，创建一个表（如果不存在），并在插入刷新的数据集之前截断该表。我使用的角色具有使用、读取、写入、创建表权限以及阶段权限，设置如下：
grant usage, read, write on future stages in schema <schema> to role <role>
我通过雪花连接器在 python 中使用 write_pandas 函数。文档说该函数使用 PUT 和 Copy Into 命令：
To write the data to the table, the function saves the data to Parquet files, uses the PUT command to upload these files to a temporary stage, and uses the COPY INTO <table> command to copy the data from the files to the table. You can use some of the function parameters to control how the PUT and COPY INTO <table> statements are executed.
我仍然收到错误消息，指出我无法对架构进行操作，并且我不确定还需要添加什么。有人拥有运行 write_pandas 命令所需的权限列表吗？
小智 6
write_pandas()不会自动创建表。如果事先不存在该表，则需要自行创建该表。每次运行时write_pandas()，它只会将数据帧附加到您指定的表中。
另一方面，如果您使用df.to_sql(..., method=pd_writer)将 pandas 数据帧写入雪花，它会自动为您创建表，并且您可以使用if_existsinto_sql()来指定不同的行为 - 追加、替换或失败 - 如果表已经存在。
我有一个相当不优雅的解决方案，可以完成表创建和附加的工作，所有这些都无需离开我的 Jupyter。
我将此代码保存在我的 sql 实用程序文件中。get_col_types 函数将创建创建表所需的列名称和数据类型的字典。
def get_col_types(df):
'''
Helper function to create/modify Snowflake tables; gets the column and dtype pair for each item in the dataframe
args:
df: dataframe to evaluate
'''
import numpy as np
# get dtypes and convert to df
ct = df.dtypes.reset_index().rename(columns={0:'col'})
ct = ct.apply(lambda x: x.astype(str).str.upper()) # case matching as snowflake needs it in uppers
# only considers objects at this point
# only considers objects and ints at this point
ct['col'] = np.where(ct['col']=='OBJECT', 'VARCHAR', ct['col'])
ct['col'] = np.where(ct['col'].str.contains('DATE'), 'DATETIME', ct['col'])
ct['col'] = np.where(ct['col'].str.contains('INT'), 'NUMERIC', ct['col'])
ct['col'] = np.where(ct['col'].str.contains('FLOAT'), 'FLOAT', ct['col'])
# get the column dtype pair
l = []
for index, row in ct.iterrows():
l.append(row['index'] + ' ' + row['col'])
string = ', '.join(l) # convert from list to a string object
string = string.strip()
return string
def create_table(table, action, col_type, df):
'''
Function to create/replace and append to tables in Snowflake
args:
table: name of the table to create/modify
action: whether do the initial create/replace or appending; key to control logic
col_type: string with column name associated dtype, each pair separated by a comma; comes from get_col_types() func
df: dataframe to load
dependencies: function get_col_types(); helper function to get the col and dtypes to create a table
'''
import pandas as pd
import snowflake.connector as snow
from snowflake.connector.pandas_tools import write_pandas
from snowflake.connector.pandas_tools import pd_writer
database=database
warehouse=warehouse
schema=schema
# set up connection
conn = snow.connect(
account = ACCOUNT,
user = USER,
password = PW,
warehouse = warehouse,
database = database,
schema = schema,
role = ROLE)
# set up cursor
cur = conn.cursor()
if action=='create_replace':
# set up execute
cur.execute(
""" CREATE OR REPLACE TABLE
""" + table +"""(""" + col_type + """)""")
#prep to ensure proper case
df.columns = [col.upper() for col in df.columns]
# write df to table
write_pandas(conn, df, table.upper())
elif action=='append':
# convert to a string list of tuples
df = str(list(df.itertuples(index=False, name=None)))
# get rid of the list elements so it is a string tuple list
df = df.replace('[','').replace(']','')
# set up execute
cur.execute(
""" INSERT INTO """ + table + """
VALUES """ + df + """
""")
工作示例：
# create df
l1 = ['cats','dogs','frogs']
l2 = [10, 20, 30]
df = pd.DataFrame(zip(l1,l2), columns=['type','age'])
col_type = get_col_types(df)
create_table('table_test', 'create_replace', col_type, df)
# now that the table is created, append to it
l1 = ['cow','cricket']
l2 = [45, 20]
df2 = pd.DataFrame(zip(l1,l2), columns=['type','age'])
append_table('table_test', 'append', None, df2)
小智 1
Windows 10、Python 3.9.4、雪花连接器-Python 2.4.2、Pandas 1.1.5
我对write_pandas函数有同样的问题。
我拥有Snowflake 的帐户管理员权限。Python 代码和错误回溯附在下面。
但是，如果我要显式编写 CSV 文件，我可以使用以下两个函数从 CSV 文件上传数据：
所以，这肯定是write_pandas函数的问题。
```import pandas as pd
```import snowflake.connector
```...
```from snowflake.connector.pandas_tools import write_pandas
```conn = snowflake.connector.connect(
``` user=strSnowflakeUserLogin,
``` password=strSnowflakeUserPassword,
``` account=strSnowflakeAccount,
``` role=strSnowflakeUserRole,
``` warehouse=strSnoflakeWarehouse,
``` database=strSnowflakeDatabase,
``` schema=strSnowflakeSchema
``` )
Traceback (most recent call last):
File "myPython.py", line xxx, in <module> myPythonModule()
write_pandas(conn, df, strSnowflakeTable)
File "C:\Users\<username>\AppData\Local\Programs\Python\Python39\lib\site-packages\snowflake\connector\pandas_tools.py", line 197, in write_pandas
copy_results = cursor.execute(copy_into_sql, _is_internal=True).fetchall()
File "C:\Users\<username>\AppData\Local\Programs\Python\Python39\lib\site-packages\snowflake\connector\cursor.py", line 692, in execute
Error.errorhandler_wrapper(
File "C:\Users\<username>\AppData\Local\Programs\Python\Python39\lib\site-packages\snowflake\connector\errors.py", line 258, in errorhandler_wrapper
cursor.errorhandler(connection, cursor, error_class, error_value)
File "C:\Users\<username>\AppData\Local\Programs\Python\Python39\lib\site-packages\snowflake\connector\errors.py", line 188, in default_errorhandler
raise error_class(
snowflake.connector.errors.ProgrammingError: 001757 (42601): SQL compilation error:
Table 'mySnowflakeTable' does not exist
```...
```write_pandas(conn, df, strSnowflakeTable)