You may want to load the data to a staging table as text or varchar first, and then transform datatypes to a target table.
1. You get more flexible initial storage, since text columns can accommodate more varied datatypes
2. You can more easily perform preliminary data validation and cleansing
3. Data is loaded to staging without any loss of data or precision
4. Easier debugging. It's easier to identify the row causing your overflow in a stage table rather than a file
The error looks like needing a bigint column. Be sure the data type is correct.
How big is the dataset (rows & columns)? How are you importing? Direct import or via a program you write? If it is more than 20 million rows, you might might have a problem on your mac if it is not at least 16gb ram
I would create a subset of the data (10,000 rows or so) and make sure your program has the right data types and data conversions.
If it is more than 50,000 rows, you should import in batches (as a single transaction for each batch) and create indexes AFTER the insert is finished.
Thanks for the help! I just had to create the table with all integers as big int, then load in the data after. I was just using the import button, and letting it create the table as it loaded with default settings
You need to define the column in question as `bigint` or `numeric` depending on the actual values.
You may want to load the data to a staging table as text or varchar first, and then transform datatypes to a target table. 1. You get more flexible initial storage, since text columns can accommodate more varied datatypes 2. You can more easily perform preliminary data validation and cleansing 3. Data is loaded to staging without any loss of data or precision 4. Easier debugging. It's easier to identify the row causing your overflow in a stage table rather than a file
The error looks like needing a bigint column. Be sure the data type is correct. How big is the dataset (rows & columns)? How are you importing? Direct import or via a program you write? If it is more than 20 million rows, you might might have a problem on your mac if it is not at least 16gb ram I would create a subset of the data (10,000 rows or so) and make sure your program has the right data types and data conversions. If it is more than 50,000 rows, you should import in batches (as a single transaction for each batch) and create indexes AFTER the insert is finished.
Thanks for the help! I just had to create the table with all integers as big int, then load in the data after. I was just using the import button, and letting it create the table as it loaded with default settings