728x90
๋ฐ˜์‘ํ˜•

2025/02/19 4

[Python][pandas] Parquet ํŒŒ์ผ ํฌ๋งท: ๊ณ ์† ๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ์— ์ตœ์ ํ™”๋œ ์ปฌ๋Ÿผ ์ €์žฅ ๋ฐฉ์‹

๋ฐ์ดํ„ฐ ๋ถ„์„๊ณผ ๋จธ์‹ ๋Ÿฌ๋‹์„ ํ•˜๋‹ค ๋ณด๋ฉด CSV, JSON, Excel ๋“ฑ์˜ ํŒŒ์ผ ํฌ๋งท์„ ์ž์ฃผ ์‚ฌ์šฉํ•˜๊ฒŒ ๋œ๋‹ค. ํ•˜์ง€๋งŒ ๋Œ€์šฉ๋Ÿ‰ ๋ฐ์ดํ„ฐ๋ฅผ ๋‹ค๋ฃฐ ๋•Œ๋Š” ์†๋„์™€ ์ €์žฅ ํšจ์œจ์„ฑ์ด ์ค‘์š”ํ•œ๋ฐ, ์ด๋Ÿด ๋•Œ Parquet ํฌ๋งท์ด ๊ฐ•๋ ฅํ•œ ๋Œ€์•ˆ์ด ๋  ์ˆ˜ ์žˆ๋‹ค.1. Parquet๋ž€?Apache Parquet๋Š” ์ปฌ๋Ÿผ ๊ธฐ๋ฐ˜ ์ €์žฅ ๋ฐฉ์‹(columnar storage format)์„ ์‚ฌ์šฉํ•˜๋Š” ์˜คํ”ˆ์†Œ์Šค ๋ฐ์ดํ„ฐ ํฌ๋งท์ž„. Hadoop ์ƒํƒœ๊ณ„์—์„œ ๊ฐœ๋ฐœ๋˜์—ˆ์œผ๋ฉฐ, ๋Œ€์šฉ๋Ÿ‰ ๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ์™€ ๋ถ„์„ ์„ฑ๋Šฅ์„ ๊ทน๋Œ€ํ™”ํ•˜๋Š” ๋ฐ ์ตœ์ ํ™”๋จ.ํŠน์ง•์ปฌ๋Ÿผ ๊ธฐ๋ฐ˜ ์ €์žฅ(Columnar Storage)CSV๋‚˜ JSON ๊ฐ™์€ ํฌ๋งท์€ ๋ฐ์ดํ„ฐ๋ฅผ ํ–‰(Row) ๋‹จ์œ„๋กœ ์ €์žฅํ•˜์ง€๋งŒ, Parquet์€ ์ปฌ๋Ÿผ(Column) ๋‹จ์œ„๋กœ ์ €์žฅํ•จํŠน์ • ์ปฌ๋Ÿผ๋งŒ ์ฝ์–ด๋„ ๋˜๋ฏ€๋กœ, ๋ถ„์„ ์†๋„๊ฐ€ ํ–ฅ์ƒ๋จ์••์ถ• ๋ฐ ์ธ์ฝ”๋”ฉ(Co..

[Learn][English] "Can I get this to go?" – ์Œ์‹ ํฌ์žฅ ์ฃผ๋ฌธํ•  ๋•Œ ์œ ์šฉํ•œ ์˜์–ด ํ‘œํ˜„

์‹๋‹น์ด๋‚˜ ์นดํŽ˜์—์„œ ์Œ์‹์„ ๋จน์ง€ ์•Š๊ณ  ํฌ์žฅํ•ด์„œ ๊ฐ€๊ณ  ์‹ถ์„ ๋•Œ ์–ด๋–ป๊ฒŒ ๋งํ•ด์•ผ ํ• ๊นŒ?๋ฐ”๋กœ "Can I get this to go?"๋ผ๋Š” ํ‘œํ˜„์„ ์‚ฌ์šฉํ•˜๋ฉด ๋œ๋‹ค.1. "Can I get this to go?"์˜ ์˜๋ฏธ"Can I get this to go?"๋Š” "์ด๊ฑฐ ํฌ์žฅํ•  ์ˆ˜ ์žˆ๋‚˜์š”?"๋ผ๋Š” ๋œป์œผ๋กœ, ์Œ์‹์„ ํฌ์žฅ ์ฃผ๋ฌธํ•  ๋•Œ ์ž์ฃผ ์‚ฌ์šฉ๋˜๋Š” ํ‘œํ˜„์ž„.์˜ˆ๋ฅผ ๋“ค์–ด, ํ–„๋ฒ„๊ฑฐ ๊ฐ€๊ฒŒ์—์„œ ์ฃผ๋ฌธํ•œ ์Œ์‹์„ ๋งค์žฅ์—์„œ ๋จน์ง€ ์•Š๊ณ  ๊ฐ€์ ธ๊ฐ€๊ณ  ์‹ถ๋‹ค๋ฉด "Can I get this to go?"๋ผ๊ณ  ๋งํ•˜๋ฉด ๋จ.2. ์˜ˆ๋ฌธ๊ณผ ํ•จ๊ป˜ ๋ฐฐ์šฐ๊ธฐ(1) ๊ธฐ๋ณธ์ ์ธ ํ™œ์šฉ ์˜ˆ๋ฌธCan I get this to go, please?→ ์ด๊ฑฐ ํฌ์žฅํ•  ์ˆ˜ ์žˆ๋‚˜์š”? I’d like to get this to go.→ ์ด๊ฑฐ ํฌ์žฅํ•ด ์ฃผ์„ธ์š”. Can I have my coffee ..

[Python][AI] ํ•œ๊ตญ ๋กœ๋˜ ๋ถ„์„: ๋‹น์ฒจ ๋ฒˆํ˜ธ ๋ถ„์„๊ณผ ํŒจํ„ด ์ฐพ๊ธฐ(EDA)

2025.02.18 - [๊ฐœ๋ฐœ Code/์ธ๊ณต์ง€๋Šฅ A.I.] - [Python][AI] ํ•œ๊ตญ ๋กœ๋˜ ๋ถ„์„: ๋‹น์ฒจ ํ™•๋ฅ ๊ณผ ์˜ˆ์ธก์˜ ๋ถˆ๊ฐ€๋Šฅ์„ฑ [Python][AI] ํ•œ๊ตญ ๋กœ๋˜ ๋ถ„์„: ๋‹น์ฒจ ํ™•๋ฅ ๊ณผ ์˜ˆ์ธก์˜ ๋ถˆ๊ฐ€๋Šฅ์„ฑ1. ํ•œ๊ตญ ๋กœ๋˜ ๊ฐœ๋…๋กœ๋˜๋Š” 1๋ถ€ํ„ฐ 45๊นŒ์ง€์˜ ์ˆซ์ž ์ค‘ 6๊ฐœ๋ฅผ ์„ ํƒํ•˜๋Š” ๋ฐฉ์‹์˜ ๋ณต๊ถŒ์ด๋‹ค. ๊ตฌ๋งค์ž๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋ฐฉ๋ฒ•์œผ๋กœ ๋ฒˆํ˜ธ๋ฅผ ์„ ํƒํ•  ์ˆ˜ ์žˆ๋‹ค.์ž๋™ ์„ ํƒ: 45๊ฐœ์˜ ๋ฒˆํ˜ธ ์ค‘ 6๊ฐœ ๋ฒˆํ˜ธ๋ฅผ ๋ฌด์ž‘์œ„๋กœ ๋ถ€์—ฌ๋ฐ›5hr1rnp.tistory.com ๋ณธ ๊ธ€์—์„œ๋Š” ๋กœ๋˜ ๋‹น์ฒจ ๋ฐ์ดํ„ฐ์— ๋Œ€ํ•œ ํƒ์ƒ‰์  ๋ฐ์ดํ„ฐ ๋ถ„์„(EDA)์„ ์ง„ํ–‰ํ•˜๋ฉฐ, ๋‹ค์–‘ํ•œ ํ†ต๊ณ„์  ํŠน์ง•์„ ์‚ดํŽด๋ณด๊ฒ ๋‹ค.1. ๋ฐ์ดํ„ฐ ์ „์ฒ˜๋ฆฌ๋กœ๋˜ ๋ฐ์ดํ„ฐ๋Š” ๋™ํ–‰๋ณต๊ถŒ ์‚ฌ์ดํŠธ์—์„œ ์ œ๊ณตํ•˜๋Š” ์—‘์…€ ํŒŒ์ผ์„ ๊ฐ๊ฐ lotto-1.xls(1~600ํšŒ), lotto-2.xls(601~1,159ํšŒ)๋กœ ์ €์žฅํ•จ..

728x90
๋ฐ˜์‘ํ˜•