sql - how to valiate Millions of data? -

-

how validate scenario?

scenario 1:

source file flat file contains millions of data. data source file loaded target table in data base.

now question how validate if data loaded in target table correctly??

note: can’t use xls validate have millions of records in it.

there lots of ways 1 can validate data. of depends on 3 things:

  1. how time have validation?

  2. what processing capabilities?

  3. is data on qa or production sql server?

if in qa , have lots of processing power, can basic checks:

  • where there warnings or errors during data load?
  • count total number of items in database vs. raw file
  • count total number of null records in database
  • check total number of columns vs. raw file
  • check length of variables. expected?
  • are character columns unexpectedly truncated?
  • are numeric columns out correct number of significant digits?
  • are dates reasonable? example, if expected dates 2004, 1970?
  • how many duplicates there?
  • check if data in columns make sense. few questions can ask: rows "shifted?" numeric variables in numeric columns? key column key? column names make sense? check of null records should detect these things.
  • can manually calculate columns , compare calculation 1 in file?

if low on processing power or on production server , not want risk degrading performance other users, can many of above checks simple random sample. take, say, 100,000 rows @ time.; or, stratify if needed.

these few checks can do. more comparisons , sanity checks, better off are.

most importantly, communicate these findings , anything seems strange file owner. should able give additional insight data load correct, or if gave right file in first place.

you're loading data , providing many reasonable checks possible. if they're satisfied outcome, , you're satisfied outcome, should consider data valid.


Comments

Post a Comment

Popular posts from this blog

Hatching array of circles in AutoCAD using c# -

-
i beginner in autocad plugins .i have created array of circle using following code. now, have hatch circles in array .how that??how hatch circles @ once? moreover, how can return hatch method , use hatch objects.sorry language.thanks in advance. [assembly: commandclass(typeof(beamsection.class1))] namespace beamsection { public class class1 { //class1 obj = new class1(); [commandmethod("beamatsupport")] public void addlightweightpolyline() { // current document , database document acdoc = application.documentmanager.mdiactivedocument; database accurdb = acdoc.database; promptpointresult ppr = acdoc.editor.getpoint("\nselect starting point "); var ucs = acdoc.editor.currentusercoordinatesystem; point3d startingpt = ppr.value.transformby(ucs); list<double> radius = new list<double>(); radius.add(0.5);
Read more

ios - UITEXTFIELD InputView Uipicker not working in swift -

-
hello trying pop uipickerview view programmatically when user clicks on textfield. have tried doesn't doing anything. isn't working. nothing happening when click textfield class userprofiletableviewcontroller: uitableviewcontroller,uitextfielddelegate,uipickerviewdelegate,uipickerviewdatasource { var itempicker: uipickerview! = uipickerview() @iboutlet weak var gendertxtfield: uitextfield! var gender = ["male","female"] override func viewdidload() { super.viewdidload() gendertxtfield.delegate = self itempicker!.delegate = self itempicker!.datasource = self itempicker!.backgroundcolor = uicolor.blackcolor() self.gendertxtfield.inputview = itempicker } func numberofcomponentsinpickerview(pickerview: uipickerview) -> int{ return 1 } // returns # of rows in each component.. func pickerview(pickerview: uipickerview, numberofrowsincomponent component:
Read more

Python Pig Latin Translator -

-
so have been working on @ codecademy.com while trying learn python. don't know if blind , can't see why won't work or what. here's code won't let me use. function print variable if had typed in , print "empty" if didn't type anything. print "welcome english pig latin translator!" original = raw_input("what's name?") if len(original) > 0 print original else print "empty" if len(original) > 0 else the above lines contains syntax errors. try: if len(original) > 0: else: note added colons, ( : ).
Read more