Subscribe to this thread
Home - General / All posts - Import Census ACS geodatabase
teckstein6 post(s)
#24-May-16 14:28

Is it possible to import the nationwide Census tract database - located here https://www.census.gov/geo/maps-data/data/tiger-data.html - directly into Manifold? I want to be able to create thematic maps of some of the basic demographic data at the tract level as well as some customize tract data that I create in Excel and import as a table into Manifold.

An alternative would be to import all the state-level shapefiles, cut and paste to build a national, census-tract map and connect tract data to that, but I was hoping to start with a nation-level census tract map to start with.

As is probably obvious, I am a beginner, so excuse me if there is an obvious answer that I am missing when reading (but likely not fully understanding) the documentation.

thanks.

Dimitri


7,413 post(s)
#24-May-16 15:47

No worries. Pick the format you want to import and then visit the user manual, going to the Import and Export book and then the Import Drawing book. Note there are topics there for importing from shapefiles, tiger line format, etc.

I see from https://www.census.gov/geo/maps-data/data/tiger-line.html they now have 2015 edition as shapefiles. You can download from their web interface at https://www.census.gov/cgi-bin/geo/shapefiles/index.php or from their FTP site.

It's mildly annoying they offer tracts by state and don't have an option for "entire US" but maybe they have that on their FTP site. I haven't been able to connect to their published FTP site (takes too long to respond).

teckstein6 post(s)
#03-Jun-16 18:35

Thanks -- I just did the shape files downloads and avoid the geodatabase hassles. Took a little more time to pull in all 50 states, but it worked.

tjhb
10,094 post(s)
#04-Jun-16 00:25

If you ever need to do this again, you could ask for help with importing all drawings and their assigning projections by script. (You might have done that this time, using a forum search perhaps.)

[Edit.] I think you meant getting the shapefiles from the Census site. If you still need to import everything to Manifold (efficiently) just ask.

And the correct FTP site for the shapefiles seems to be ftp://ftp.census.gov/geo/tiger/TIGER2015/. (The links on the Census pages point to ftp://ftp2..., which currently times out as Dimitri has said.)

With this ftp address you could use a batch download tool like DownThemAll to queue up a whole folder at a time.

tjhb
10,094 post(s)
#04-Jun-16 00:48

I see DownThemAll might be a bit dead now* due to changes in the Firefox extensions architecture. I haven't used it for a while. It could be worth making a temporary installation of an old version of Firefox for this task in my opinion, it saves so much human time. I have archived ESR versions back to 2011 if anyone finds them hard to get hold of. Obviously there would be security risks in installing an old version, especially in theory. Or there might be better alternatives I don't know of.

[Added.] I should have doublechecked. DownThemAll is alive. Version 2.0.19, released April 2016, "works with Firefox 41.0 and later". No need for an old version of anything. Version history.

Dimitri


7,413 post(s)
#04-Jun-16 06:39

I still cannot connect to the amended FTP URL... that, too, times out for me.

Another option for an FTP client is Filezilla, which can also download a whole folder hierarchy at a time.

hugh
200 post(s)
#04-Jun-16 02:40

another source is the National Historical Geographic Information System (https://www.nhgis.org/). However what you did is more direct and easier if what you need are the most recent tract boundaries without much count data.

If you want to do a check and compare, I put the NHGIS tract boundaries I am using now here on dropbox (may not stay there too long however):

https://dl.dropboxusercontent.com/u/44755972/UStract2014NHGIS.map 701 MB

sorry I did not think of doing this sooner if I could have saved you some work :(

Dimitri


7,413 post(s)
#04-Jun-16 07:02

Thanks for posting! Just downloaded it.

antoniocarlos

609 post(s)
#26-May-16 19:46

If you download anything from the US Census as an ESRI file geodatabase. Manifold will not be able to read it directly. I use QGIS for reading these.


How soon?

RonHendrickson
283 post(s)
#27-May-16 03:21

Not sure what you mean, the last time (<1 yr) I used the US Census geodatabases (shapefiles), Manifold swallowed them up easily. No problem.

antoniocarlos

609 post(s)
#27-May-16 04:06

Shapefiles are one thing. File Geodatabases are a different beast. The US Census publishes some data as Zipped File Geodatabases that as far as I can tell can not be read by Manifold currently. Thankfully it also produces shapefiles.

Here are some examples of File Geodatabases

http://geo/maps-data/data/tiger-geodatabases.html

Here is some light reading about file geodatabases.

http://resources.arcgis.com/en/help/main/10.1/index.html#//003n00000007000000

Cheers


How soon?

Dimitri


7,413 post(s)
#27-May-16 08:31

File Geodatabases are a different beast.

True, but why would you cause yourself extra trouble by downloading a fragile format like a file geodatabase when you have the choice of downloading shapefiles? That goes for either Manifold or QGIS.

After all, shapefiles are the native format of QGIS. As the QGIS documentation puts it: "The standard vector file format used in QGIS is the ESRI shapefile."

QGIS is good at working with shapefiles, so even if you are bent on using QGIS it would be better to use the shapefiles.

I note that even ESRI does not recommend the use of the file geodatabase API that is incorporated into the GDAL/OGR distribution utilized by QGIS. As ESRI puts it: "This API does not replace ArcObjects as the recommended approach to interacting with the geodatabase." Good advice. :-)

antoniocarlos

609 post(s)
#27-May-16 11:55

I agree!

But the Census has chosen to do this for large datasets that are not well served by the shapefile format as a convenience for people that can read the file format. For example, it publishes all the US census block groups as one big file geodatabase but as far as I can tell tf does not do that in one big shapefile.

The QGIS team has hacked the file format and is able to read some forms of it.

But your advise is correct D. Download shapefiles and avoid File Geodatabases if you can.

Cheers


How soon?

Dimitri


7,413 post(s)
#27-May-16 15:08

The QGIS team has hacked the file format and is able to read some forms of it.

Why don't they just use the ESRI API? It's free to download and use - no need to reverse engineer the format. :-)

hugh
200 post(s)
#28-May-16 22:23

I think they did via GDAL which used the Esri API to make the driver. Still limited though since the the Esri API does not work for rasters, as far as I know you need an Esri license to access those. But no problem for vector census data. On the latter I think the reason why they don't use shapefiles for datasets with lots of content is the 255 field limit in dbf files that are part of shapefiles.

Dimitri


7,413 post(s)
#29-May-16 08:10

why they don't use shapefiles for datasets with lots of content is the 255 field limit in dbf files that are part of shapefiles.

And also a factor is the gross limit on size for shapefiles. File geodatabases can be much bigger. But that doesn't affect most vector data sets that Census publishes because those tend to be small and fit within the limits of shapefiles.

2GB of vector data is bigger than most desktop applications can handle so Census tends to distribute vector data sets in much smaller pieces. Shapefiles are fine for those.

QGIS does indeed use GDAL which incorporates ESRI's API for file geodatabases and which indeed does not include support for rasters in file geodatabases. That is not a factor for census block groups since those are vector data and not rasters. But by way of discussion I can understand why there might have been some confusion over whether the API supports rasters or not.

When ESRI's API for the file geodabase was announced in 2010 it was noted the API was provided with limitations and that the "initial release" would not support rasters.

More recently (for example in March, 2016) presentations on the API are almost word-for-word identical to the initial discussion of limitations but do not mention "initial release" and simply say the API "does not" support rasters. The exact language is:

---------

File Geodatabase API / Features Not Supported

While the File Geodatabase API supports reading the schema and data of complex geodatabase types, the API does not honor geodatabase behavior on inserts, deletes or updates to the following dataset types:

- Annotation and Dimension feature classes

- Relationship Classes

- Networks (GN and ND)

- Topologies

- Terrains

- Representations

- Parcel Fabrics

File Geodatabase API / Features Not Supported

- Raster Datasets, Raster Catalogs, Mosaic Datasets and Raster Attributes are not supported.

- Spatial queries are limited to the envelope-intersects operator.

- Attachments are not be [sic] supported

- Joins are not supported

An earlier description, still applicable, includes this: "The API will not prevent users from attempting to edit objects with complex behavior, the onus will be on the developer to understand what they should and should not edit through the API and avoid editing datasets that have geodatabase behavior."

---------

For rasters, the preferred path can be inferred from the official description of the API: "This API does not replace ArcObjects as the recommended approach to interacting with the geodatabase."

Therefore if you want to work with raster data sets in file geodatabases you use ArcObjects, which is a commercial product offered by ESRI and not a free API.

To branch off into a different direction I respectfully note that there was a lot of happy talk around the original issuance of the file geodatabase API as being intended to make it freely available to anyone who wanted to code an application that could "open File Geodatabase tables in non-ESRI applications to view or modify data." But the actual license, as more alert people at the time noted, did not allow that. It was surprisingly restrictive in many ways, for example, for years blocking use by ESRI competitors.

The good news is that ESRI has changed the license for the API to what is one of the best and most straightforward licenses around. Now the license for the API does indeed open the API for use by anyone, including ESRI competitors. Kudos to ESRI for moving to genuinely and professionally opening up the API for use by anyone. It will be in Radian and 9, for example. Good work ESRI.

Unfortunately the raster bit is not in the API. But what is in there already is very useful and good for the world GIS community.

hugh
200 post(s)
#30-May-16 01:55

"Census tends to distribute vector data sets in much smaller pieces"

However for tabulation datasets such as for blockgroups there can be huge number of fields, and I often hit a 1600 field limit importing into postgresSQL. Following antoniocarlos's suggestion, I just read and processed 1,685 fields from file GDB x25housing data using QGIS. Thank you!

It will be great to have this capability in Manifold and Radian.

tjhb
10,094 post(s)
#30-May-16 03:58

Isn't normal form suppose to save us from completely bonkers tables like this? Who could find a table with 1,685 fields useful? That should be, say, 50 tables with a common ID field.

Or is there a good reason, for this data?

hugh
200 post(s)
#30-May-16 06:06

The census form itself has relatively few questions -- for example age and income would be two fields. However for reasons of confidentiality only tabulated results are given in smaller areas like tracts and block groups. Thus age might be broken down into tabulations by 5 year intervals and income by $5K intervals, so there could be a field containing the count of persons in a tract between 40 and 45 yrs with income between $30-35K.

However you are correct, there are better ways to organize the databases. That's what companies do with the free data from census and charge for their efforts.

jkelly


1,234 post(s)
#01-Jun-16 03:26

For what it's worth, similar happens with the free Australian data as well. Maybe there is a reason beyond users using Excel only, but I've yet to see one.

Incredibly frustrating if you want to extract useful data and store in a database because you have to convert the star schema back into relational, when I'd say it came from relational in the first place.


James Kelly

http://www.locationsolve.com

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.