Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Commitdf77c32

Browse files
committed
Added load_data.py example
1 parent717de29 commitdf77c32

File tree

2 files changed

+63
-0
lines changed

2 files changed

+63
-0
lines changed
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
id,text
2+
1,"Python was conceived in the late 1980s[40] by Guido van Rossum at Centrum Wiskunde & Informatica (CWI) in the Netherlands as a successor to the ABC programming language, which was inspired by SETL,[41] capable of exception handling and interfacing with the Amoeba operating system.[10] Its implementation began in December 1989.[42] Van Rossum shouldered sole responsibility for the project, as the lead developer, until 12 July 2018, when he announced his ""permanent vacation"" from his responsibilities as Python's ""benevolent dictator for life"", a title the Python community bestowed upon him to reflect his long-term commitment as the project's chief decision-maker.[43] In January 2019, active Python core developers elected a five-member Steering Council to lead the project.[44][45]"
3+
2,"Python was conceived in the late 1980s[40] by Guido van Rossum at Centrum Wiskunde & Informatica (CWI) in the Netherlands as a successor to the ABC programming language, which was inspired by SETL,[41] capable of exception handling and interfacing with the Amoeba operating system.[10] Its implementation began in December 1989.[42] Van Rossum shouldered sole responsibility for the project, as the lead developer, until 12 July 2018, when he announced his ""permanent vacation"" from his responsibilities as Python's ""benevolent dictator for life"", a title the Python community bestowed upon him to reflect his long-term commitment as the project's chief decision-maker.[43] In January 2019, active Python core developers elected a five-member Steering Council to lead the project.[44][45]"
4+
3,"Python 2.0 was released on 16 October 2000, with many major new features such as list comprehensions, cycle-detecting garbage collection, reference counting, and Unicode support.[46] Python 3.0, released on 3 December 2008, with many of its major features backported to Python 2.6.x[47] and 2.7.x. Releases of Python 3 include the 2to3 utility, which automates the translation of Python 2 code to Python 3.[48]"
5+
4,"Python 2.7's end-of-life was initially set for 2015, then postponed to 2020 out of concern that a large body of existing code could not easily be forward-ported to Python 3.[49][50] No further security patches or other improvements will be released for it.[51][52] Currently only 3.8 and later are supported (2023 security issues were fixed in e.g. 3.7.17, the final 3.7.x release[53]). While Python 2.7 and older is officially unsupported, a different unofficial Python implementation, PyPy, continues to support Python 2, i.e. ""2.7.18+"" (plus 3.9 and 3.10), with the plus meaning (at least some) ""backported security updates"".[54]"
6+
5,"In 2021 (and again twice in 2022), security updates were expedited, since all Python versions were insecure (including 2.7[55]) because of security issues leading to possible remote code execution[56] and web-cache poisoning.[57] In 2022, Python 3.10.4 and 3.9.12 were expedited[58] and 3.8.13, because of many security issues.[59] When Python 3.9.13 was released in May 2022, it was announced that the 3.9 series (joining the older series 3.8 and 3.7) would only receive security fixes in the future.[60] On 7 September 2022, four new releases were made due to a potential denial-of-service attack: 3.10.7, 3.9.14, 3.8.14, and 3.7.14.[61][62]"
7+
6,"As of October 2023, Python 3.12 is the stable release, and 3.12 and 3.11 are the only versions with active (as opposed to just security) support. Notable changes in 3.11 from 3.10 include increased program execution speed and improved error reporting.[63]"
8+
7,"Python 3.12 adds syntax (and in fact every Python since at least 3.5 adds some syntax) to the language, the new (soft) keyword type (recent releases have added a lot of typing support e.g. new type union operator in 3.10), and 3.11 for exception handling, and 3.10 the match and case (soft) keywords, for structural pattern matching statements. Python 3.12 also drops outdated modules and functionality, and future versions will too, see below in Development section."
9+
8,"Python 3.11 claims to be between 10 and 60% faster than Python 3.10, and Python 3.12 adds another 5% on top of that. It also has improved error messages, and many other changes."
10+
9,"Since 27 June 2023, Python 3.8 is the oldest supported version of Python (albeit in the 'security support' phase), due to Python 3.7 reaching end-of-life.[64]"
11+
10,"Python is a multi-paradigm programming language. Object-oriented programming and structured programming are fully supported, and many of their features support functional programming and aspect-oriented programming (including metaprogramming[65] and metaobjects).[66] Many other paradigms are supported via extensions, including design by contract[67][68] and logic programming.[69]"
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
importasyncio
2+
frompgmlimportCollection,Pipeline
3+
importpandasaspd
4+
5+
6+
# Initialize Collection
7+
collection=Collection("load_data_demo")
8+
9+
# Iniitalize Pipeline
10+
pipeline=Pipeline(
11+
"v1",
12+
{
13+
"text": {
14+
"splitter": {"model":"recursive_character"},
15+
"semantic_search": {"model":"intfloat/e5-small"},
16+
}
17+
},
18+
)
19+
20+
asyncdefinit_collection():
21+
awaitcollection.add_pipeline(pipeline)
22+
23+
24+
defload_documents():
25+
# This can be any loading function. For our case, we will be loading in a CSV
26+
# The important piece is that our upsert_documents wants an array of dictionaries
27+
data=pd.read_csv("./data/example_data.csv")
28+
returndata.to_dict("records")
29+
30+
31+
asyncdefmain():
32+
# We only ever need to add a Pipeline once
33+
awaitinit_collection()
34+
35+
# Get our documents. Documents are just dictionaries with at least the `id` key
36+
# E.G. {"id": "document_one, "text": "here is some text"}
37+
documents=load_documents()
38+
39+
# This does the actual uploading of our documents
40+
# It handles uploading in batches and guarantees that any documents uploaded are
41+
# split and embedded according to our Pipeline definition above
42+
awaitcollection.upsert_documents(documents)
43+
44+
# The default batch size is 100, but we can override that if we have thousands or
45+
# millions of documents to upload it will be faster with a larger batch size
46+
awaitcollection.upsert_documents(documents, {"batch_size":1000})
47+
48+
# Now we can search over our collection or do whatever else we want
49+
# See other examples for more information on searching
50+
51+
52+
asyncio.run(main())

0 commit comments

Comments
 (0)

[8]ページ先頭

©2009-2025 Movatter.jp