failure family

data quality and comparability

40 claims across 17 works.

• inconsistent industry terminology in Form PF: The SEC has not standardized the disclosures required in Form PF, and there is evidence that Form PF requirements rest on an inconsistent use of indus
• stale filed disclosure data: Under both the bankruptcy and the systemic risk disclosure regimes, filed data carries a serious risk of being out of date and less accurate at the ti
• unreliable systemic risk data: Form PF disclosures have not been standardized, and anecdotal evidence indicates that the SEC and the FSOC may be working with contradictory, misleadi
• generic data fails to reveal investor motives: Because systemic risk disclosures are far more generic and are not tailored to any specific distressed investment, importing them into bankruptcy woul
• generic and outdated data yields no improvement: Using generic and possibly outdated systemic risk data in the bankruptcy process would not improve hedge funds' bankruptcy practices in the near term.
• garbage-in-policy-out: If advisers' allegations that Form PF disclosures cannot be answered other than by guessing are correct, then the SEC's capacity to evaluate the data
• incomparable-performance-metrics: Form PF fund performance metrics are not accurate or comparable across filers, because reporting entities employ different calculation methodologies t
• document-recording-gap: Where coded categories such as cooperating, disclosure, and internal review fall well short of 100 percent of the sample, the shortfall may reflect a
• Reliance concentrated on weakest data: The SEC data collected from private fund advisers feeds every stage of the FSOC's systemic risk assessment, and the FSOC leans most heavily on precise
• Upstream data error propagates to risk assessment: Accuracy and consistency problems in the SEC's private fund data collection can impair the FSOC's ability to evaluate the systemic risk posed by priva
• Mandated data collection creates evaluation problems: Prior studies and anecdotal evidence indicate that the data collection mandated by Form PF could itself create problems for the FSOC when it evaluates
• Inconsistent adviser responses: The SEC itself reports that the consistency of investment advisers' responses on Form PF is not ensured and may be questionable.
• Non comparable filing methodologies: Advisers take different approaches and make different assumptions when completing Form PF, which the SEC identifies as a further challenge to the usab
• Filer error anomalies: The SEC's initial analysis of Form PF data turned up anomalies attributed to filer error, which prompted SEC concern about the quality of the informat
• Remedies insufficient to restore confidence: Expanding the uses of Form PF data remains difficult so long as there is insufficient confidence in the accuracy of what advisers report, notwithstand
• Acknowledged data gaps: The FSOC itself conceded that available data was insufficient when it tried to identify the activities of the twenty largest United States fund manage
• Defective inputs to stage one screen: Because several core Form PF questions feeding the FSOC's stage one threshold screen are themselves defective, the FSOC's systemic risk assessment pro
• Weakest data drives interconnectedness finding: The Form PF counterparty questions most affected by filer interpretation, Questions 22 and 23, are the very ones the FSOC uses in stage two to determi
• Estimates and assumptions substitute for measurement: Widespread filer disagreement with Form PF definitions implies that a large share of filers are uncertain how to answer, which raises the possibility
• Error propagation from filings to designation: If the FSOC relies on inaccurate Form PF data in its systemic risk assessment, its work on private funds may itself be erroneous.
• Reporting issues affect assessment: Private fund advisers reporting under Form PF encountered issues that could affect the FSOC's systemic risk assessment, but the author does not claim
• Defect to use matching reveals assessment risk: Matching the identified Form PF defects against the FSOC's specific uses of that data suggests possible inaccuracies in the FSOC's systemic risk asses
• pre-crisis data gap: Any conclusion that hedge funds contributed to the financial crisis of 2007-2008 is circumstantial or anecdotal, because the data needed to test it, o
• Deal flow data unavailability: Venture capital deal flow, meaning the totality of potential deals and business plans screened by venture capitalists, would provide the optimal asses
• Unobtainable counterparty exposure data: Some of the most sensitive Form PF disclosures are not readily obtainable by the funds themselves: counterparty credit exposure often cannot be determ
• Undeterminable counterparty exposure data: Form PF required disclosures of counterparty credit exposure constitute sensitive information that individual fund managers often cannot readily deter
• No comparative yardstick: Three features make it uniquely challenging for retail investors to evaluate the risks of unconstrained mutual funds: the lack of standard benchmarks,
• Human error in manual fee calculation: Manual per transaction fee calculation and settlement is prone to human error, and these errors are removed through the use of blockchain technology.
• Unanswerable disclosure items: Some Form PF disclosure requirements are not answerable as designed, because counterparty credit exposure is sensitive information that individual pri
• Inaccurate regulatory data undermines systemic risk assessment: The SEC's private fund data collection encountered accuracy and consistency problems that hampered the FSOC's ability to evaluate the systemic risk of
• Regulator leans hardest on the weakest data items: The FSOC relied most heavily on some of the most problematic disclosure items the SEC collects, even though SEC data played a crucial role at every st
• Error propagation from filer uncertainty to regulator conclusions: If the FSOC relies on Form PF data that is subject to inaccuracies, because uncertain filers complete the form using estimates and assumptions, then t
• big data validity illusion: Increasing the quantity of data does not dissolve the foundational methodological problems of data: construct validity, measurement, reliability, and
• Recursive human verification problem: Human shortcomings in micro task work such as limited attention span, irrationality, and inaccuracy create a need for verification, but manual verific
• big-data-validity-deficit: Big data is often not the output of instruments designed to generate valid and reliable data suitable for scientific analysis, so foundational data qu
• Manual Verification Regress: Human shortcomings in micro task work such as limited attention span, irrationality, and inaccuracy create verification requirements, but manual verif
• connectivity and legal constriction of supply: Privacy rules, copyright, and the uneven global distribution of digital connectivity independently reduce both the availability and the diversity of h
• data staleness: In fast-moving fields such as technology, law, and public policy, models trained on obsolete datasets fail to capture new trends, behaviors, or regula
• annotation quality drift: Maintaining consistent annotation quality across many annotators and automated systems is unsolved at scale, and small labeling errors translate into
• opaque-and-ambiguous-precedent: Existing precedent regimes fail on accessibility and weighting: national precedent is scattered across opaque reporters, paywalled databases and untra