Intake für die Datenwissenschaften#
Intake erleichtert das Laden vieler verschiedener Formate und Typen. Um einen vollständigen Überblick zu erhalten, schaut euch das Plugin Directory und das Intake Project Dashboard an. Intake überführt die Daten dann in übliche Speicherformate wie Pandas DataFrames, Numpy-Arrays oder Python-Listen. Anschließend sind sie leicht durchsuchbar und auch für verteilte Systeme zugänglich. Sollte euch ein Plugin fehlen, könnt ihr auch selbst welche estellen, wie in Making Drivers beschrieben.
Laden einer Datenquelle#
Im Folgenden werden wir zwei csv-Datensätze lesen und in einen Intake-Katalog überführen.
[1]:
import intake
ds = intake.open_csv('states_*.csv')
print(ds)
<intake.source.csv.CSVSource object at 0x7fc5575ccd50>
Mit der open_*-Funktion von Intake lassen sich verschiedenen Datenquellen einlesen. Je nach Datenformat oder Dienst lassen sich unterschiedliche Argmuente verwenden.
Konfigurieren des Suchpfades für Datenquellen#
Intake überprüft die Intake-Konfigurationsdatei nach catalog_patund die Umgebungsvariable "INTAKE_PATH" auf eine durch Doppelpunkte getrennte Liste von Pfaden bzw. Semikolon in Windows, um nach Katalogdateien zu suchen. Beim Import intake werden alle Einträge aus allen Katalogen angezeigt, auf die als Teil eines globalen Katalogs von intake.cat referenziert werden.
Daten lesen#
Intake liest Daten in Container verschiedener Formate:
Tabellen in Pandas DataFrames
Mehrdimensionale Array in Numpy Arrays
Semistrukturierte Daten in Python-Listen von Objekten, üblicherweise Dictionaries
Um herauszufinden, in welchem Containerformat Intake die Daten vorhält, könnt ihr das container-Attribut verwenden:
[2]:
ds.container
[2]:
'dataframe'
Neben dataframe kann das Ergebnis auch ndarray oder python sein.
[3]:
df = ds.read()
df.head()
[3]:
| state | slug | code | nickname | website | admission_date | admission_number | capital_city | capital_url | population | population_rank | constitution_url | state_flag_url | state_seal_url | map_image_url | landscape_background_url | skyline_background_url | twitter_url | facebook_url | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Alabama | alabama | AL | Yellowhammer State | http://www.alabama.gov | 1819-12-14 | 22 | Montgomery | http://www.montgomeryal.gov | 4833722 | 23 | http://alisondb.legislature.state.al.us/alison... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alab... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alabamagov | https://www.facebook.com/alabamagov |
| 1 | Alaska | alaska | AK | The Last Frontier | http://alaska.gov | 1959-01-03 | 49 | Juneau | http://www.juneau.org | 735132 | 47 | http://www.legis.state.ak.us/basis/folioproxy.... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alas... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alaska | https://www.facebook.com/AlaskaLocalGovernments |
| 2 | Arizona | arizona | AZ | The Grand Canyon State | https://az.gov | 1912-02-14 | 48 | Phoenix | https://www.phoenix.gov | 6626624 | 15 | http://www.azleg.gov/Constitution.asp | https://cdn.civil.services/us-states/flags/ari... | https://cdn.civil.services/us-states/seals/ari... | https://cdn.civil.services/us-states/maps/ariz... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | NaN | NaN |
| 3 | Arkansas | arkansas | AR | The Natural State | http://arkansas.gov | 1836-06-15 | 25 | Little Rock | http://www.littlerock.org | 2959373 | 32 | http://www.arkleg.state.ar.us/assembly/Summary... | https://cdn.civil.services/us-states/flags/ark... | https://cdn.civil.services/us-states/seals/ark... | https://cdn.civil.services/us-states/maps/arka... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/arkansasgov | https://www.facebook.com/Arkansas.gov |
| 4 | California | california | CA | Golden State | http://www.ca.gov | 1850-09-09 | 31 | Sacramento | http://www.cityofsacramento.org | 38332521 | 1 | http://www.leginfo.ca.gov/const-toc.html | https://cdn.civil.services/us-states/flags/cal... | https://cdn.civil.services/us-states/seals/cal... | https://cdn.civil.services/us-states/maps/cali... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/cagovernment | NaN |
[4]:
for chunk in ds.read_chunked(): print('Chunk: %d' % len(chunk))
Chunk: 24
Chunk: 26
[5]:
ddf = ds.to_dask()
ddf.head()
[5]:
| state | slug | code | nickname | website | admission_date | admission_number | capital_city | capital_url | population | population_rank | constitution_url | state_flag_url | state_seal_url | map_image_url | landscape_background_url | skyline_background_url | twitter_url | facebook_url | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Alabama | alabama | AL | Yellowhammer State | http://www.alabama.gov | 1819-12-14 | 22 | Montgomery | http://www.montgomeryal.gov | 4833722 | 23 | http://alisondb.legislature.state.al.us/alison... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alab... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alabamagov | https://www.facebook.com/alabamagov |
| 1 | Alaska | alaska | AK | The Last Frontier | http://alaska.gov | 1959-01-03 | 49 | Juneau | http://www.juneau.org | 735132 | 47 | http://www.legis.state.ak.us/basis/folioproxy.... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alas... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alaska | https://www.facebook.com/AlaskaLocalGovernments |
| 2 | Arizona | arizona | AZ | The Grand Canyon State | https://az.gov | 1912-02-14 | 48 | Phoenix | https://www.phoenix.gov | 6626624 | 15 | http://www.azleg.gov/Constitution.asp | https://cdn.civil.services/us-states/flags/ari... | https://cdn.civil.services/us-states/seals/ari... | https://cdn.civil.services/us-states/maps/ariz... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | NaN | NaN |
| 3 | Arkansas | arkansas | AR | The Natural State | http://arkansas.gov | 1836-06-15 | 25 | Little Rock | http://www.littlerock.org | 2959373 | 32 | http://www.arkleg.state.ar.us/assembly/Summary... | https://cdn.civil.services/us-states/flags/ark... | https://cdn.civil.services/us-states/seals/ark... | https://cdn.civil.services/us-states/maps/arka... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/arkansasgov | https://www.facebook.com/Arkansas.gov |
| 4 | California | california | CA | Golden State | http://www.ca.gov | 1850-09-09 | 31 | Sacramento | http://www.cityofsacramento.org | 38332521 | 1 | http://www.leginfo.ca.gov/const-toc.html | https://cdn.civil.services/us-states/flags/cal... | https://cdn.civil.services/us-states/seals/cal... | https://cdn.civil.services/us-states/maps/cali... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/cagovernment | NaN |
[6]:
cat = intake.open_catalog('us_states.yml')
[7]:
list(cat)
[7]:
['states']
[8]:
cat.states.to_dask()[['state','slug']].head()
[8]:
| state | slug | |
|---|---|---|
| 0 | Alabama | alabama |
| 1 | Alaska | alaska |
| 2 | Arizona | arizona |
| 3 | Arkansas | arkansas |
| 4 | California | california |
[9]:
cat.states(csv_kwargs={'header': None, 'skiprows': 1}).read().head()
[9]:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Alabama | alabama | AL | Yellowhammer State | http://www.alabama.gov | 1819-12-14 | 22 | Montgomery | http://www.montgomeryal.gov | 4833722 | 23 | http://alisondb.legislature.state.al.us/alison... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alab... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alabamagov | https://www.facebook.com/alabamagov |
| 1 | Alaska | alaska | AK | The Last Frontier | http://alaska.gov | 1959-01-03 | 49 | Juneau | http://www.juneau.org | 735132 | 47 | http://www.legis.state.ak.us/basis/folioproxy.... | https://cdn.civil.services/us-states/flags/ala... | https://cdn.civil.services/us-states/seals/ala... | https://cdn.civil.services/us-states/maps/alas... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/alaska | https://www.facebook.com/AlaskaLocalGovernments |
| 2 | Arizona | arizona | AZ | The Grand Canyon State | https://az.gov | 1912-02-14 | 48 | Phoenix | https://www.phoenix.gov | 6626624 | 15 | http://www.azleg.gov/Constitution.asp | https://cdn.civil.services/us-states/flags/ari... | https://cdn.civil.services/us-states/seals/ari... | https://cdn.civil.services/us-states/maps/ariz... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | NaN | NaN |
| 3 | Arkansas | arkansas | AR | The Natural State | http://arkansas.gov | 1836-06-15 | 25 | Little Rock | http://www.littlerock.org | 2959373 | 32 | http://www.arkleg.state.ar.us/assembly/Summary... | https://cdn.civil.services/us-states/flags/ark... | https://cdn.civil.services/us-states/seals/ark... | https://cdn.civil.services/us-states/maps/arka... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/arkansasgov | https://www.facebook.com/Arkansas.gov |
| 4 | California | california | CA | Golden State | http://www.ca.gov | 1850-09-09 | 31 | Sacramento | http://www.cityofsacramento.org | 38332521 | 1 | http://www.leginfo.ca.gov/const-toc.html | https://cdn.civil.services/us-states/flags/cal... | https://cdn.civil.services/us-states/seals/cal... | https://cdn.civil.services/us-states/maps/cali... | https://cdn.civil.services/us-states/backgroun... | https://cdn.civil.services/us-states/backgroun... | https://twitter.com/cagovernment | NaN |