Google BigQuery REST API C# copy/export table with nested schema as CSV -
is there way export whole table nested schema google bigquery using rest api csv?
there example doing (https://cloud.google.com/bigquery/docs/exporting-data) not nested schema. works fine on not nested columns in table. here code of part:
pagedenumerable<tabledatalist, bigqueryrow> result2 = client.listrows(datasetid, result.reference.tableid); stringbuilder sb = new stringbuilder(); foreach (var row in result2) { sb.append($"{row["visitorid"]}, {row["visitnumber"]}, {row["totals.hits"]}{environment.newline}"); } using (var stream = new memorystream(encoding.utf8.getbytes(sb.tostring()))) { var obj = gcsclient.uploadobject(bucketname, filename, contenttype, stream); }
in bq there columns totals.hits, totals.visits...if try address them got errormessage there not such column. if addressing "totals" objectname "system.collections.generic.dictionary`2[system.string,system.object]" in rows in csv.
is there possibility that? in end want table ga in bq csv somewhere else.
it possible. select every column need in following shema und flatten need flattened.
string query = $@" #legacysql select visitorid, visitnumber, visitid, visitstarttime, date, hits.hitnumber hitnumber, hits.product.productsku product.productsku flatten(flatten({tablename},hits),hits.product)"; //creating job query , activating legacy sql bigqueryjob job = client.createqueryjob(query, new createqueryjoboptions { uselegacysql = true }); bigqueryresults queryresult = client.getqueryresults(job.reference.jobid, new getqueryresultsoptions()); stringbuilder sb = new stringbuilder(); //getting headers ga table , write them first row of new table int count = 0; (int = 0; <= queryresult.schema.fields.count() - 1; i++) { string columenname = ""; var header = queryresult.schema.fields[0].name; if (i + 1 >= queryresult.schema.fields.count) columenname = queryresult.schema.fields[i].name; else columenname = queryresult.schema.fields[i].name + ","; sb.append(columenname); } //getting data ga table , write them row row new table sb.append(environment.newline); foreach (var row in queryresult.getrows()) { count++; if (count % 1000 == 0) console.writeline($"item {count} finished"); int blub = queryresult.schema.fields.count; (int64 j = 0; j < convert.toint64(blub); j++) { try { if (row.rawrow.f[convert.toint32(j)] != null) sb.append(row.rawrow.f[convert.toint32(j)].v + ","); } catch (exception) { } } sb.append(environment.newline); }
Comments
Post a Comment