Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitc02b3e3

Browse files
committed
Added offline summarization example
1 parent2259e51 commitc02b3e3

File tree

5 files changed

+124
-13
lines changed

5 files changed

+124
-13
lines changed

‎pgml-sdks/pgml/python/examples/README.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
#Examples
22

33
##Prerequisites
4-
Before running any examples first install dependencies and set theDATABASE_URL environment variable:
4+
Before running any examples first install dependencies and set thePGML_DATABASE_URL environment variable:
55
```
66
pip install -r requirements.txt
7-
exportDATABASE_URL={YOUR DATABASE URL}
7+
exportPGML_DATABASE_URL={YOUR DATABASE URL}
88
```
99

1010
Optionally, configure a .env file containing a DATABASE_URL variable.
@@ -29,3 +29,6 @@ This is an example to find documents relevant to a question from the collection
2929

3030
##[Load Data](./load_data.py)
3131
This is a simple example to show best practices for upserting data to a collection.
32+
33+
##[Offline Summarization](./offline_summarization.py)
34+
This sample shows how to perform summarization over documents that have been added to a collection using SQL.
Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
1-
id,text
2-
1,"Python was conceived in the late 1980s[40] by Guido van Rossum at Centrum Wiskunde & Informatica (CWI) in the Netherlands as a successor to the ABC programming language, which was inspired by SETL,[41] capable of exception handling and interfacing with the Amoeba operating system.[10] Its implementation began in December 1989.[42] Van Rossum shouldered sole responsibility for the project, as the lead developer, until 12 July 2018, when he announced his ""permanent vacation"" from his responsibilities as Python's ""benevolent dictator for life"", a title the Python community bestowed upon him to reflect his long-term commitment as the project's chief decision-maker.[43] In January 2019, active Python core developers elected a five-member Steering Council to lead the project.[44][45]"
3-
2,"Python was conceived in the late 1980s[40] by Guido van Rossum at Centrum Wiskunde & Informatica (CWI) in the Netherlands as a successor to the ABCprogramming language, which was inspired by SETL,[41] capable of exception handling and interfacing with the Amoeba operating system.[10] Itsimplementation began in December 1989.[42] Van Rossum shouldered sole responsibility fortheproject, as the lead developer, until 12 July 2018, when he announced his ""permanent vacation"" from his responsibilities as Python's ""benevolent dictator for life"", a title the Python community bestowed upon him to reflect his long-term commitment as the project's chief decision-maker.[43] In January 2019, active Python core developers elected a five-member Steering Council to lead the project.[44][45]"
4-
3,"Python 2.0 was released on 16 October 2000, with many major new features such as list comprehensions, cycle-detecting garbage collection, reference counting, and Unicode support.[46] Python 3.0, released on 3 December 2008, with many of its major features backported to Python 2.6.x[47] and 2.7.x. Releases of Python 3 include the 2to3 utility, which automates the translation of Python 2 code to Python 3.[48]"
5-
4,"Python 2.7's end-of-life was initially set for 2015, then postponed to 2020 out of concern that a large body of existing code could not easily be forward-ported to Python 3.[49][50] No further security patches or other improvements will be released for it.[51][52] Currently only 3.8 and later are supported (2023 security issues were fixed in e.g. 3.7.17, the final 3.7.x release[53]). While Python 2.7 and older is officially unsupported, a different unofficial Python implementation, PyPy, continues to support Python 2, i.e. ""2.7.18+"" (plus 3.9 and 3.10), with the plus meaning (at least some) ""backported security updates"".[54]"
6-
5,"In 2021 (and again twice in 2022), security updates were expedited, since all Python versions were insecure (including 2.7[55]) because of security issues leading to possible remote code execution[56] and web-cache poisoning.[57] In 2022, Python 3.10.4 and 3.9.12 were expedited[58] and 3.8.13, because of many security issues.[59] When Python 3.9.13 was released in May 2022, it was announced that the 3.9 series (joining the older series 3.8 and 3.7) would only receive security fixes in the future.[60] On 7 September 2022, four new releases were made due to a potential denial-of-service attack: 3.10.7, 3.9.14, 3.8.14, and 3.7.14.[61][62]"
7-
6,"As of October 2023, Python 3.12 is the stable release, and 3.12 and 3.11 are the only versions with active (as opposed to just security) support. Notable changes in 3.11 from 3.10 include increased program execution speed and improved error reporting.[63]"
8-
7,"Python 3.12 adds syntax (and in fact every Python since at least 3.5 adds some syntax) to the language, the new (soft) keyword type (recent releases have added a lot of typing support e.g. new type union operator in 3.10), and 3.11 for exception handling, and 3.10 the match and case (soft) keywords, for structural pattern matching statements. Python 3.12 also drops outdated modules and functionality, and future versions will too, see below in Development section."
9-
8,"Python 3.11 claims to be between 10 and 60% faster than Python 3.10, and Python 3.12 adds another 5% on top of that. It also has improved error messages, and many other changes."
10-
9,"Since 27 June 2023, Python 3.8 is the oldest supported version of Python (albeit in the 'security support' phase), due to Python 3.7 reaching end-of-life.[64]"
11-
10,"Python is a multi-paradigm programming language. Object-oriented programming and structured programming are fully supported, and many of their features support functional programming and aspect-oriented programming (including metaprogramming[65] and metaobjects).[66] Many other paradigms are supported via extensions, including design by contract[67][68] and logic programming.[69]"
1+
id,url,text
2+
1,"https://en.wikipedia.org/wiki/Python_(programming_language)","Python was conceived in the late 1980s[40] by Guido van Rossum at Centrum Wiskunde & Informatica (CWI) in the Netherlands as a successor to the ABC programming language, which was inspired by SETL,[41] capable of exception handling and interfacing with the Amoeba operating system.[10] Its implementation began in December 1989.[42] Van Rossum shouldered sole responsibility for the project, as the lead developer, until 12 July 2018, when he announced his ""permanent vacation"" from his responsibilities as Python's ""benevolent dictator for life"", a title the Python community bestowed upon him to reflect his long-term commitment as the project's chief decision-maker.[43] In January 2019, active Python core developers elected a five-member Steering Council to lead the project."
3+
2,"https://en.wikipedia.org/wiki/Python_(programming_language)","Python is a high-level, general-purposeprogramming language. Itsdesign philosophy emphasizes code readability withtheuse of significant indentation."
4+
3,"https://en.wikipedia.org/wiki/Python_(programming_language)","Python 2.0 was released on 16 October 2000, with many major new features such as list comprehensions, cycle-detecting garbage collection, reference counting, and Unicode support.[46] Python 3.0, released on 3 December 2008, with many of its major features backported to Python 2.6.x[47] and 2.7.x. Releases of Python 3 include the 2to3 utility, which automates the translation of Python 2 code to Python 3.[48]"
5+
4,"https://en.wikipedia.org/wiki/Python_(programming_language)","Python 2.7's end-of-life was initially set for 2015, then postponed to 2020 out of concern that a large body of existing code could not easily be forward-ported to Python 3.[49][50] No further security patches or other improvements will be released for it.[51][52] Currently only 3.8 and later are supported (2023 security issues were fixed in e.g. 3.7.17, the final 3.7.x release[53]). While Python 2.7 and older is officially unsupported, a different unofficial Python implementation, PyPy, continues to support Python 2, i.e. ""2.7.18+"" (plus 3.9 and 3.10), with the plus meaning (at least some) ""backported security updates"".[54]"
6+
5,"https://en.wikipedia.org/wiki/Python_(programming_language)","In 2021 (and again twice in 2022), security updates were expedited, since all Python versions were insecure (including 2.7[55]) because of security issues leading to possible remote code execution[56] and web-cache poisoning.[57] In 2022, Python 3.10.4 and 3.9.12 were expedited[58] and 3.8.13, because of many security issues.[59] When Python 3.9.13 was released in May 2022, it was announced that the 3.9 series (joining the older series 3.8 and 3.7) would only receive security fixes in the future.[60] On 7 September 2022, four new releases were made due to a potential denial-of-service attack: 3.10.7, 3.9.14, 3.8.14, and 3.7.14.[61][62]"
7+
6,"https://en.wikipedia.org/wiki/Python_(programming_language)","As of October 2023, Python 3.12 is the stable release, and 3.12 and 3.11 are the only versions with active (as opposed to just security) support. Notable changes in 3.11 from 3.10 include increased program execution speed and improved error reporting.[63]"
8+
7,"https://en.wikipedia.org/wiki/Python_(programming_language)","Python 3.12 adds syntax (and in fact every Python since at least 3.5 adds some syntax) to the language, the new (soft) keyword type (recent releases have added a lot of typing support e.g. new type union operator in 3.10), and 3.11 for exception handling, and 3.10 the match and case (soft) keywords, for structural pattern matching statements. Python 3.12 also drops outdated modules and functionality, and future versions will too, see below in Development section."
9+
8,"https://en.wikipedia.org/wiki/Python_(programming_language)","Python 3.11 claims to be between 10 and 60% faster than Python 3.10, and Python 3.12 adds another 5% on top of that. It also has improved error messages, and many other changes."
10+
9,"https://en.wikipedia.org/wiki/Python_(programming_language)","Since 27 June 2023, Python 3.8 is the oldest supported version of Python (albeit in the 'security support' phase), due to Python 3.7 reaching end-of-life.[64]"
11+
10,"https://en.wikipedia.org/wiki/Python_(programming_language)","Python is a multi-paradigm programming language. Object-oriented programming and structured programming are fully supported, and many of their features support functional programming and aspect-oriented programming (including metaprogramming[65] and metaobjects).[66] Many other paradigms are supported via extensions, including design by contract[67][68] and logic programming.[69]"

‎pgml-sdks/pgml/python/examples/load_data.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
11
importasyncio
22
frompgmlimportCollection,Pipeline
33
importpandasaspd
4+
fromdotenvimportload_dotenv
45

6+
load_dotenv()
57

68
# Initialize Collection
79
collection=Collection("load_data_demo")
@@ -17,6 +19,7 @@
1719
},
1820
)
1921

22+
2023
asyncdefinit_collection():
2124
awaitcollection.add_pipeline(pipeline)
2225

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
frompgmlimportCollection,Pipeline
2+
importpsycopg2
3+
importasyncio
4+
importpandasaspd
5+
fromdotenvimportload_dotenv
6+
importos
7+
8+
load_dotenv()
9+
db_url=os.environ['PGML_DATABASE_URL']
10+
11+
# Initialize Collection
12+
collection=Collection("summary_demo")
13+
14+
# Iniitalize Pipeline
15+
pipeline=Pipeline(
16+
"v1",
17+
{
18+
"text": {
19+
"splitter": {"model":"recursive_character"},
20+
"semantic_search": {"model":"intfloat/e5-small"},
21+
}
22+
},
23+
)
24+
25+
26+
asyncdefinit_collection():
27+
awaitcollection.add_pipeline(pipeline)
28+
29+
30+
defload_documents():
31+
# This can be any loading function. For our case, we will be loading in a CSV
32+
# The important piece is that our upsert_documents wants an array of dictionaries
33+
data=pd.read_csv("./data/example_data.csv")
34+
returndata.to_dict("records")
35+
36+
asyncdefmain():
37+
# We only ever need to add a Pipeline once
38+
awaitinit_collection()
39+
40+
# Get our documents. Documents are just dictionaries with at least the `id` key
41+
# E.G. {"id": "document_one, "text": "here is some text"}
42+
documents=load_documents()
43+
44+
# This does the actual uploading of our documents
45+
# It handles uploading in batches and guarantees that any documents uploaded are
46+
# split and embedded according to our Pipeline definition above
47+
awaitcollection.upsert_documents(documents)
48+
49+
# Now that we have the documents in our database, let's do some summarization
50+
conn=psycopg2.connect(db_url)
51+
cur=conn.cursor()
52+
53+
# First lets create our summary table
54+
# This table has three columns:
55+
# id - auto incrementing primary key
56+
# document_id - the document id we summarized
57+
# summary - the summary of the text of the document
58+
# version - the document `text` key version. This is really just
59+
# the hash of the `text` column. Stored in the version column of the documents table.
60+
# We store this column so we don't recompute summaries
61+
cur.execute("""
62+
CREATE TABLE IF NOT EXISTS summary_demo.summaries (
63+
id SERIAL PRIMARY KEY,
64+
document_id INTEGER REFERENCES summary_demo.documents (id) UNIQUE,
65+
summary TEXT,
66+
version VARCHAR(32)
67+
)
68+
""")
69+
conn.commit()
70+
71+
# Now let's fill up our summary table
72+
# This query is very efficient as it only updates the summary for documents not currently in
73+
# the table, or whos `text` key has changed since the last summary
74+
cur.execute("""
75+
INSERT INTO summary_demo.summaries (document_id, summary, version)
76+
SELECT
77+
sdd.id, (
78+
SELECT transform[0]->>'summary_text' FROM pgml.transform(
79+
task => '{
80+
"task": "summarization",
81+
"model": "google/pegasus-xsum"
82+
}'::JSONB,
83+
inputs => ARRAY[
84+
sdd.document->>'text'
85+
]
86+
)
87+
),
88+
sdd.version->'text'->>'md5'
89+
FROM summary_demo.documents sdd
90+
LEFT OUTER JOIN summary_demo.summaries as sds ON sds.document_id = sdd.id
91+
WHERE sds.document_id IS NULL OR sds.version != sdd.version->'text'->>'md5'
92+
ON CONFLICT (document_id) DO UPDATE SET version = EXCLUDED.version, summary = EXCLUDED.summary
93+
""")
94+
conn.commit()
95+
96+
# Let's see what our summaries are
97+
cur.execute("SELECT * FROM summary_demo.summaries")
98+
forrowincur.fetchall():
99+
print(row)
100+
101+
# Purposefully not removing the tables and collection so they can be inspected in the database
102+
103+
104+
asyncio.run(main())

‎pgml-sdks/pgml/python/examples/requirements.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
aiohttp==3.8.5
22
aiosignal==1.3.1
33
async-timeout==4.0.3
4+
psycopg2==2.9.9
45
attrs==23.1.0
56
certifi==2023.7.22
67
charset-normalizer==3.2.0

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp